MauveDB: Statistical Modeling of Data inside Relational Databases

Sensor networks almost by definition present complex issues related to data quality. This is especially true for networks of mobile devices because of higher potential for data loss and errors. Moreover, the physical context, especially location, plays a very important role for mobile devices and must be considered when trying to interpret the sensor readings. The traditional approach to dealing with these issues is to incorporate a statistical model of the real-world phenomenon into the processing of sensor data; models can help in providing more robust interpretations of sensor readings, by accounting for spatial and temporal biases, by identifying faults, and by allowing estimation of missing values or future states. We illustrate this through two representative sensor data processing tasks. Re-gridding using interpolation or regression models: Sensor data, especially that generated by mobile sensor networks, is typically incomplete because of sensor node failures and communication loss. Such data must be gap-filled and possibly re-gridded into a uniform time-space dataset that would be easier to visualize and analyze (Figure 1). A variety of regression or interpolation models can be used for the purpose. Inferring hidden variables: For many deployments, the variables of interest may not be directly observable using sensors, but must be inferred from the sensed data using a model of the physical process. The inference task could simply be estimating the true position of a mobile device from noisy GPS data (using Kalman Filters), or detecting failures (using an HMM), or could be as complex as inferring the transportation mode of a user given just the noisy location information [3]. Models like dynamic Bayesian networks can be used for this purpose. Unfortunately today’s data management systems do not support such statistical modeling tasks natively, forcing the users to use external tools for this purpose. This process is time-consuming, leads to much repetition of functionality and more importantly, requires familiarity with advanced data analysis tools like Matlab or R. Our research is motivated by the question of how to make it easier for a user or an application developer to apply simple statistical models to data, preferably using declarative interfaces ? In particular, can we push this process inside a database system, so that the existing database tools like data cubes, that can be immensely useful in visualizing enormous multi-dimensional datasets, can be used analyzing sensor data ?